Machine learning-based diagnostic classifier

ABSTRACT

Systems and methods for utilizing machine learning to generate a trans-diagnostic classifier that is operative to concurrently diagnose a plurality of different mental health disorders using a single trans-diagnostic questionnaire that includes a plurality of questions (e.g., 17 questions). Machine learning techniques are used to process labeled training data to build statistical models that include trans-diagnostic item-level questions as features to create a screen to classify groups of subjects as either healthy or as possibly having a mental health disorder. A subset of questions are selected from the multiple self-administered mental health questionnaires and used to autonomously screen subjects across multiple mental health disorders without physician involvement, optionally remotely and repeatedly, in a short amount of time.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/400,312, entitled “Machine Learning-Based Diagnostic Classifier,”filed on May 1, 2019, which claims priority under 35 U.S.C. § 119 toU.S. Provisional Application No. 62/665,243, entitled, “MachineLearning-Based Diagnostic Classifier,” filed May 1, 2018, the contentsof which are incorporated herein by reference in their entirety.

FIELD

The present disclosure generally relates to medical diagnostic tools,and more particularly, to systems and methods for machine learning-basedmental health diagnostic tools.

BACKGROUND

Mental health screening and diagnosis requires a time-consuminginterview between patients and highly-trained specialists within aclinic. Currently available remotely-administered self-assessments tendto be based on discrete diagnostic categories that may fail to revealtrans-diagnostic or sub-clinical behavioral changes that warrantintervention.

SUMMARY

The various examples of the present disclosure are directed towardssystems and methods for screening the mental health of patients. In afirst embodiment, an exemplary system includes a display, a microphone,a camera, a memory, and a control system. The camera is positioned tocapture an image in front of the display and configured to output videodata. The memory contains machine readable medium comprising machineexecutable code and has stored instructions for performing a method ofevaluating the mental health of a user. The control system is coupled tothe memory, includes one or more processors, and executes the machineexecutable code. This causes the control system to perform the followingseries of steps.

The control system executes a test application, upon receiving, from theuser interface, an indication to initiate a test. The control systemterminates the test application upon receiving an indication to stop thetest. The test application includes (1) displaying, on the display, aseries of questions from mental health questionnaires comprising textand answers for each question, (2) displaying, on the display, livevideo data recorded by the camera, (3) recording, by the camera, a setof test video data, (4) recording, by the microphone, a set of testaudio data, (5) receiving, though the user interface, an answer for eachof the series of questions to yield a selection of answers, and (6)processing, using a machine learning model, the selection of answers,the set of test video data, and the set of audio data to output a mentalhealth indication of the user.

In some examples, the indication to stop the test application is adetermination, by the control system, that a user face is not within animage captured by the camera.

In some examples, recording, by the microphone, includes initiating therecording upon determining, by the control system, that the user isspeaking.

In some examples, the control system is configured to perform additionalsteps, including receiving the set of test video data and the set oftest audio data. The received set of test video data is preprocessed toidentify a plurality of video segments, each video segment correspondingto one question in the series of questions and comprising a time window.The received set of test audio data is preprocessed to identify aplurality of audio segments, each audio segment corresponding to onequestion in the series of questions and comprising a time window.

In some examples, the plurality of audio segments and the plurality ofvideo segments are preprocessed to identify overlapping time windows.The control system outputs a set of integrated audio and video segmentsbased on the identified overlapping time windows.

In some examples, the machine learning model is any of a generalizedlinear model, a regression model, a logistical regression model, and/ora supervised machine learning classification model.

In some examples, the machine learning model is a generalized linearmodel generated by performing a series of steps. The steps provide forreceiving labeled training data for a plurality of individuals. Thelabeled training data includes (1) indications whether each of theplurality of individuals has one or more mental health disorders, (2)audio and video data recorded for each of the plurality of individualsrecording during a training test, and (3) a selection of answers to thequestionnaires from each of the plurality of individuals. The steps thenprovide for determining a plurality of features from the labeledtraining data and training an initial machine learning model in asupervised manner, based on the plurality of features. The steps thenprovide for extracting importance measures for each of the plurality offeatures, based on the training of the initial machine learning model. Aplurality of subset machine learning models is then generated based onthe extracted importance measures for the plurality of features. Aclassification performance of the generated plurality of subset machinelearning models is then evaluated; and based on the evaluation, at leastone of the subset machine learning models is selected as the generalizedlinear model.

In some examples, the mental health indication identifies a likelihoodof the user having one of a plurality of mental health disorders,including a neuropsychiatric disorder, schizophrenia, and/or a bipolardisorder. In some examples, the mental health indication identifieswhether the user is a patient or a healthy control.

A second embodiment of the present disclosure provides a system,including a display, a microphone, a camera, a memory, and a controlsystem. The camera is positioned to capture an image in front of thedisplay and configured to output video data. The memory contains machinereadable medium comprising machine executable code and has storedinstructions for performing a method of evaluating the mental health ofa user. The control system is coupled to the memory, includes one ormore processors, and executes the machine executable code. This causesthe control system to perform the following series of steps.

The control system executes a test application, upon receiving, from theuser interface, an indication to initiate a test. The control systemterminates the test application upon receiving an indication to stop thetest. The test application provides for (1) displaying text on thedisplay for the user to read, (2) recording, by the camera, a set oftest video data during the test, (3) displaying, on the display, awindow displaying live video data recorded by the camera, (4)iteratively processing the set of test video data during the test, (5)recording, by the microphone, a set of test audio data during the test,and (6) processing the set of test audio data and test video data toidentify audio and video features and storing the audio and videofeatures in the memory. The continual processing step provides foridentifying a face of the user, and determining whether each of aplurality of pixels of the face are within a frame. If the face isoutside the frame, the processing step provides for stopping the test.

In some examples, the displayed text comprises a series of questionsfrom mental health questionnaires including text and answers for eachquestion.

In some examples, each of the audio and video features correspond to aquestion in the series of questions.

Additional examples of the second embodiment are provided for as withrespect to the first embodiment.

A third embodiment of the present disclosure provides a system forscreening the mental health of patients, the system includes a memoryand a control system. The memory contains machine readable mediumcomprising machine executable code and has stored instructions forperforming a method of evaluating the mental health of a user. Thecontrol system is coupled to the memory, includes one or moreprocessors, and executes the machine executable code. This causes thecontrol system to (1) receive a set of answer data representing answersfrom a user to a series of questions from mental health questionnaires,(2) receive a set of test video data recorded during a test representingthe face of the user while the user is reading text, (3) process the setof test video data to output a set of video features, (4) receive a setof test audio data recorded during the test representing the voice ofthe user while the user is reading text, (5) process the set of audiodata to output a set of audio features, and (6) process, using a machinelearning model, the set of answer data, the set of video features, andthe set of audio features to output an indication of the mental healthof the user. In some examples, the machine learning model is any of: ageneralized linear model, a regression model, a logistical regressionmodel, and/or a supervised machine learning classification model.Additional embodiments of the third embodiment are as provided for abovewith respect to the first and second embodiments.

A fourth embodiment of the present disclosure provides machine learningtraining system. The system includes at least one non-transitoryprocessor-readable storage medium and at least one processor. Thestorage medium stores at least one of processor-executable instructionsor data. The processor is communicatively coupled to the at least onenon-transitory processor-readable storage medium. In operation, the atleast one processor is configured to receive labeled training data. Thetraining data includes data for a plurality of individuals thatindicates whether each of the plurality of individuals has one or moreof a plurality of mental health disorders. The training dataadditionally includes (1) answers to mental health questionnaires, and(2) video data and audio data. The mental health questionnaires wereadministered to the plurality of individuals. The video data and audiodata were recorded while each of the plurality of individuals read textfrom a digital display. The video data is processed to identify portionsof the video data comprising the face of the individual, and the audiodata is processed to identify sounds representing the voice of theindividual. The processor is further configured to process the answers,the audio data, and the video data to output a plurality of features.The processor then trains an initial machine learning model in asupervised manner based on the received training data. The processorthen extracts an importance measure for each of the plurality offeatures from the trained initial machine learning model. The processorthen generates a plurality of subset machine learning models based onthe extracted importance measures for the plurality of features. Theprocessor then evaluates a classification performance of the generatedplurality of subset machine learning models. The processor then selectsat least one of the plurality of subset machine learning models as adiagnostic classifier. The processor then stores the features of thediagnostic classifier in the at least one non-transitoryprocessor-readable storage medium for subsequent use as a screeningtool.

In some examples, the selected subset machine learning model includes aportion of the plurality of features, the portion selected from featureshaving an importance measure above a threshold value.

In some examples, at least twenty features of the plurality of featureshave an importance measure above the threshold value, and the portionincludes at least ten features and less than twenty features.

In some examples, each of the subset machine learning models includes adifferent selection of the portion of the plurality of features.

In some examples, the diagnostic classifier outputs a mental healthindication identifying an individual as healthy or as having a generalmental health issue.

In some examples, the diagnostic classifier outputs a mental healthindication identifying an individual as healthy or as having a specificmental health issue.

In some examples, the diagnostic classifier outputs a mental healthindication identifying an individual as having either a first specificmental health disorder or a second specific mental health disorder.

In some examples, the diagnostic classifier outputs a mental healthindication identifying a risk of developing a mental health disorder foran individual.

In some examples, the labeled training data further includes, for eachindividual in the plurality of individuals, an indication of at leastone of the following: whether the individual is healthy, whether theindividual has a general mental health issue, whether the individual hasone or more specific mental health disorders, whether the individual isat risk of developing a general mental health issue, and/or whether theindividual is at risk of developing one or more specific mental healthdisorders.

In some examples, training the initial machine learning model includesusing k-fold cross validation with logistic regression.

In some examples, each of the subset machine learning models includes adifferent combination of the plurality of features.

In some examples, the labeled training data includes at least one offunctional measurement data or physiological measurement data.

In some examples, the fourth embodiment provides for using the featuresof the diagnostic classifier as a screening tool to assess at least oneof intermediate or end-point outcomes in at least one clinical trialtesting for treatment responses.

The above summary is not intended to represent each embodiment or everyaspect of the present disclosure. Rather, the foregoing summary merelyprovides an example of some of the novel aspects and features set forthherein. The above features and advantages, and other features andadvantages of the present disclosure, will be readily apparent from thefollowing detailed description of representative embodiments and modesfor carrying out the present invention, when taken in connection withthe accompanying drawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not necessarily drawn to scale, and some ofthese elements may be arbitrarily enlarged and positioned to improvedrawing legibility. Further, the particular shapes of the elements asdrawn, are not necessarily intended to convey any information regardingthe actual shape of the particular elements, and may have been solelyselected for ease of recognition in the drawings.

FIG. 1 is a network diagram illustrating an example environment in whicha system for training and implementing a machine learning-basedtrans-diagnostic classifier may be configured, initiated and operated,according to one non-limiting illustrated implementation of the presentdisclosure.

FIG. 2 is a block diagram of an example computing system suitable forexecuting an embodiment of a machine learning-based trans-diagnosticclassifier in configured manners.

FIG. 3 is a flow diagram for a method of operating a trans-diagnosticclassifier system according to one illustrated implementation of thepresent disclosure.

FIG. 4 is a graph that shows the receiver operating characteristics(ROC) curve for an initial machine learning classifier, according to onenon-limiting illustrated implementation of the present disclosure.

FIG. 5 is a graph that shows the area under the ROC curve for aplurality of subset machine learning models that include from onefeature up to 578 features, according to one non-limiting illustratedimplementation of the present disclosure.

FIG. 6 is a graph that shows an accuracy score for the plurality ofsubset machine learning models, according to one non-limitingillustrated implementation of the present disclosure.

FIG. 7A provides an exemplary system, according to an embodiment of thepresent disclosure.

FIGS. 7B-7D show exemplary methodologies for receiving and analyzingdata, according to an embodiment of the present disclosure.

FIG. 8 shows an exemplary methodology of processing audio and videodata, according to an embodiment of the present disclosure.

FIG. 9 shows an exemplary methodology for analyzing input with a machinelearning model, according to an embodiment of the present disclosure.

FIGS. 10A-10B show exemplary user interfaces on a smart phone, accordingto an embodiment of the present disclosure.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedimplementations. However, one skilled in the relevant art will recognizethat implementations may be practiced without one or more of thesespecific details, or with other methods, components, materials, etc. Inother instances, well-known structures associated with computer systems,server computers, and/or communications networks have not been shown ordescribed in detail to avoid unnecessarily obscuring descriptions of theimplementations.

Unless the context requires otherwise, throughout the specification andclaims that follow, the word “comprising” is synonymous with“including,” and is inclusive or open-ended (i.e., does not excludeadditional, unrecited elements or method acts).

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrases “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more implementations.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contextclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contextclearly dictates otherwise.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theimplementations.

This specification describes systems and methods of screening people formental health disorders by using a machine learning approach to selectthe most informative questions from a broad set of questionnaires forassessing those disorders. These screens are fast, highly-accurate, andrely only on self-reporting by any individual (they do not need to beadministered by a mental health professional). Several screens can becreated based on the main goal of the screen, such as: 1) atrans-diagnostic screen (to determine if individual is healthy or has ageneral mental health issue), 2) a specific-disorder screen (todetermine if individual is healthy or has a specific mental healthdisorder like schizophrenia or ADHD), or 3) a differential-diagnosisscreen (to determine if individual has one specific mental healthdisorder or another specific mental health disorder like schizophreniarather than ADHD). This approach is not limited to predicting only asubset of specific mental health disorders, as it may be applied topredicting mental health issues, symptoms, or behavioral dimensions inmental health or other disorders (e.g., depression in Parkinson'sdisease, psychosis in epilepsy, dementia in multiple sclerosis (White etal., 2012)). This approach is also not limited to prediction of mentalhealth problems only using questions/questionnaires as input features,as functional (e.g., smartphone user interactions) or physiologicaltypes of measurements (e.g., magnetic resonance imaging,electroencephalography, magnetoencephalography, electrocorticography,positron emission tomography, single-photon emission computedtomography) can also provide an informative set of predictive featuresto select from for efficient and accurate mental health screening.

This approach outperforms other trans-diagnostic screens based only on asingle questionnaire (e.g., Kessler et al., 2002; Kessler et al., 2003)which highlights the advantage of taking the novel approach of combiningand selecting a subset of questions from across multiple questionnairesusing a machine learning approach. Such an approach may also helpidentify individuals who may not have a mental health disorder but maybe at risk at developing one (e.g., by identifying misclassifications ofthe model and building additional models to label them as a separategroup of at-risk individuals).

As these screens are quick and rely on self-reported answers, this setof screens could be administered from a phone, tablet, or computer app(e.g., mobile app, web browser app), with the collected data processedon the local device or in a cloud-computing environment, and transmittedwith the individual's consent to a primary care physician or a mentalhealth professional.

There is a myriad of applications that may use these screens. Thefollowing lists several non-limiting examples of applications in whichthe screens of the present disclosure may be used. The screens can beused by any individual to assess if they may have a mental healthdisorder. The screens can be used to estimate incidence and prevalenceof mental health issues in a given population (e.g., state, national,homeless, military, schools, ethnic, etc.). The screens can be used toassess both intermediate and end-point outcomes in clinical trialstesting for treatment responses. The screens can be used as a primarycare screening tool for patients with expected mental health issues toreduce inefficient and unnecessary referrals. The screens can be used toquickly triage patients suspected of mental health issues in emergencydepartment settings. The screens can be used to check the likelihood ofa self-reported disorder by an individual enrolled in a study recruitingindividuals with mental health disorders but not able to have aphysician assessment of the disorder. The screens can be used in theworkplace as it has been found that identifying and treating mentalillness is cheaper than lost productivity to companies (Kessler et al.,2009).

A machine learning system may be summarized as including at least onenon-transitory processor-readable storage medium that stores at leastone of processor-executable instructions or data; and at least oneprocessor communicatively coupled to the at least one non-transitoryprocessor-readable storage medium, in operation, the at least oneprocessor: receives labeled training data that includes data for aplurality of individuals that indicates whether each of the individualshas one or more of a plurality of mental health disorders, the labeledtraining data further including item-level responses of at least some ofthe individuals to multiple self-administered mental healthquestionnaires that each comprise one or more features; trains aninitial machine learning model in a supervised manner based at least inpart on the received training data; extracts an importance measure foreach of the plurality of features in the multiple self-administeredmental health questionnaires from the trained initial machine learningmodel; generates a plurality of subset machine learning models based atleast in part on the extracted importance measures for the plurality offeatures; evaluates the classification performance of the generatedplurality of subset machine learning models; selects at least one of thesubset machine learning models as a diagnostic classifier; and storesthe features of the diagnostic classifier in the at least onenon-transitory processor-readable storage medium for subsequent use as ascreening tool. The selected subset machine learning model may include Mof the most important N features as determined by the importancemeasures, wherein M is an integer between 10 and 20 and N is an integergreater than 20. The diagnostic classifier may be operative to determinewhether an individual has one of a plurality of mental health disorders.

The selected subset machine learning model may include at least a subsetof the following features: “I like to please other people as much as Ican”; “There are often times when I am so restless that it is impossiblefor me to sit still”; “My mood often changes, from happiness to sadness,without my knowing why”; “Although there are things that I enjoy doingby myself, I usually seem to have more fun when I do things with otherpeople”; “I am more sentimental than most people”; “I love to excel ateverything I do”; “People consider me a rather freewheeling andspontaneous person”; “I feel that I never really get all that I needfrom people”; “In unfamiliar surroundings, I am often so assertive andsociable that I surprise myself”; “I like to think about things for along time before I make a decision”; “Sometimes ideas and insights cometo me so fast that I cannot express them all”; “I have many hobbies”; “Ilike to keep my problems to myself”; “It is difficult for me to keep thesame interests for a long time because my attention often shifts tosomething else”; “How often do you have trouble wrapping up the finaldetails of a project, once the challenging parts have been done”; “Ilike to go slow in starting work, even if it is easy to do”; and“Usually I am more worried than most people that something might gowrong in the future.” In operation, the at least one processor may trainthe initial machine learning model using k-fold cross validation withlogistic regression. Each of the subset machine learning models mayinclude a different combination of the features of the initial machinelearning model. Each of the subset machine learning models may include adifferent number of the most important features of the initial machinelearning model determined by the importance measures. As would bereadily understood by one skilled in the art, variations of thesequestions can be used in the disclosed systems and methods as well. Invarious examples of the present disclosure, some additional questionscan be used, replacement/alternate questions can be used, or some of thequestions can be omitted.

One or more implementations of the present disclosure are directed tosystems and methods for utilizing machine learning to generate atrans-diagnostic classifier that is operative to concurrently diagnose aplurality of different mental health disorders using a singletrans-diagnostic questionnaire that includes a plurality of questions(e.g., 17 questions), also referred to herein as features. Generally,the inventors of the present disclosure have implemented machinelearning techniques to develop a quick, trans-diagnostic,self-administered mental health screen, which is automatically scored,to overcome at least some of the barriers noted above. It is noted thatalthough the examples discussed below include questions/questionnairesas input features for explanatory purposes, it should be appreciatedthat the systems and methods disclosed herein are not limited toprediction of mental health problems only using questions/questionnairesas input features, as functional (e.g., smartphone user interactions) orphysiological types of measurements (e.g., magnetic resonance imaging,electroencephalography, magnetoencephalography, electrocorticography,positron emission tomography, single-photon emission computedtomography) can also provide an informative set of predictive featuresto select from for efficient and accurate mental health screening.Further, the implementations discussed herein may be used to provide atrans-diagnostic screen, a specific-disorder screen, adifferential-diagnosis screen, or other types of screens.

As discussed further below, machine learning techniques may be used toprocess labeled training data to build statistical models that includetrans-diagnostic item-level questions as features to create a screen toclassify groups of subjects as either healthy or as possibly having amental health disorder. The labeled training data may include data for aplurality of individuals that indicates whether each of the individualshas one or more of a plurality of disorders, such as, but not limitedto, schizophrenia, bipolar disorder, or attention deficit andhyperactivity disorder (ADHD). For each of the individuals, the labeledtraining data also includes item-level responses to multipleself-administered mental health questionnaires (e.g., fivequestionnaires, 10 questionnaires, 20 questionnaires).

Using machine learning techniques, a subset of the questions, e.g.,15-20 questions out of more than 20 questions (e.g., 200 questions, 600questions), from the multiple self-administered mental healthquestionnaires may be selected and used to autonomously screen subjectsacross multiple mental health disorders without physician involvement,optionally remotely and repeatedly, in a short amount of time (e.g.,less than 5 minutes). The various features of the implementations of thepresent disclosure are discussed further below with reference to thefigures.

FIG. 1 is a network diagram illustrating an example environment in whicha system for generating and implementing a trans-diagnostic classifier(TDC) system 100 may be configured and initiated. In particular, anembodiment of the TDC system 100 is shown executing on one or morecomputing systems 102, including in the illustrated embodiment tooperate in an online manner and provide one or more interfaces 104(e.g., graphical user interface (GUI), applications programminginterfaces (API)) to enable one or more remote users of client computingsystems 106 to interact over one or more intervening computer networks108 with the TDC system 100 to generate, modify, and use one or moretrans-diagnostic classifiers.

Using client computing systems 106, one or more users (e.g.,researchers, physicians, patients) may interact over the computernetwork 108 with the TDC system 100 to generate a TDC and to use agenerated TDC to screen for a plurality of mental health disorders. Inat least some implementations, one or more systems may be used togenerate a classifier, and one or more different systems may be used toimplement the classifier as a screening tool. The TDC system 100 mayinclude a TDC controller component 110 (e.g., one or more processors), aTDC data storage component 112 (e.g., one or more non-transitoryprocessor-readable storage media), the interfaces 104, and other TDCcomponents 114 (e.g., processors, data storage, wired/wirelessinterfaces, input/output devices). In the illustrated example, the TDCdata storage component 112 stores labeled training data 116, one or moreinitial machine learning models 118, one or more subset machine learningmodels 120, and one or more output trans-diagnostic classifiers 122 thatmay be used to screen subjects for a plurality of mental healthdisorders. Each of these components is discussed below.

The network 108 may, for example, be a publicly accessible network oflinked networks, possibly operated by various distinct parties, such asthe Internet, with the TDC system 100 available to any users or onlycertain users over the network 108. In other embodiments, the network108 may be a private network, such as, for example, a corporate oruniversity network that is wholly or partially inaccessible tonon-privileged users. In still other embodiments, the network 108 mayinclude one or more private networks with access to and/or from theInternet. Thus, while the TDC system 100 in the illustrated embodimentis implemented in an online manner to support various users over the oneor more computer networks 108, in other embodiments a copy of the TDCsystem 100 may instead be implemented in other manners, such as tosupport a single user or a group of related users (e.g., a company orother organization), such as if the one or more computer networks 108are instead an internal computer network of the company or otherorganization, and with such a copy of the TDC system optionally notbeing available to other users external to the company or otherorganizations. The online version of the TDC system 100 and/or localcopy version of the TDC system may in some embodiments and situationsoperate in a fee-based manner, such that the one or more users providevarious fees to use various operations of the TDC system, such as to usethe TDC system 100 to screen one or more individuals for mental healthdisorders. In addition, the TDC system 100, and/or each of itscomponents, may include software instructions that execute on one ormore computing systems (not shown) by one or more processors (notshown), such as to configure those processors and computing systems tooperate as specialized machines with respect to performing theirprogrammed functionality.

FIG. 2 shows an example processor-based device 204 suitable forimplementing various embodiments described herein. For example, theprocessor-based device 204 may be representative of the computing system102 or one of the client computing systems 106 of FIG. 1 . Although notrequired, some portion of the embodiments will be described in thegeneral context of processor-executable instructions or logic, such asprogram application modules, objects, or macros being executed by one ormore processors. Those skilled in the relevant art will appreciate thatthe described embodiments, as well as other embodiments, can bepracticed with various processor-based system configurations, includinghandheld devices, such as smartphones and tablet computers, wearabledevices, multiprocessor systems, microprocessor-based or programmableconsumer electronics, personal computers (“PCs”), network PCs,minicomputers, mainframe computers, and the like.

The processor-based device 204 may, for example, take the form of aserver computer, cloud-based computing system, desktop computer,smartphone or tablet computer, which includes one or more processors206, a system memory 208 and a system bus 210 that couples varioussystem components including the system memory 208 to the processor(s)206. The processor-based device 204 will at times be referred to in thesingular herein, but this is not intended to limit the embodiments to asingle system, since in certain embodiments, there will be more than onesystem or other networked computing device involved. Non-limitingexamples of commercially available systems include, but are not limitedto, ARM processors from a variety of manufactures, Core microprocessorsfrom Intel Corporation, U.S.A., PowerPC microprocessor from IBM, Sparcmicroprocessors from Sun Microsystems, Inc., PA-RISC seriesmicroprocessors from Hewlett-Packard Company, 68xxx seriesmicroprocessors from Motorola Corporation.

The processor(s) 206 may be any logic processing unit, such as one ormore central processing units (CPUs), microprocessors, digital signalprocessors (DSPs), application-specific integrated circuits (ASICs),field programmable gate arrays (FPGAs), etc. Unless described otherwise,the construction and operation of the various blocks shown in FIG. 2 areof conventional design. As a result, such blocks need not be describedin further detail herein, as they will be understood by those skilled inthe relevant art.

The system bus 210 can employ any known bus structures or architectures,including a memory bus with memory controller, a peripheral bus, and alocal bus. The system memory 208 includes read-only memory (“ROM”) 212and random access memory (“RAM”) 214. A basic input/output system(“BIOS”) 216, which can form part of the ROM 212, contains basicroutines that help transfer information between elements withinprocessor-based device 204, such as during start-up. Some embodimentsmay employ separate buses for data, instructions and power.

The processor-based device 204 may also include one or more solid statememories, for instance Flash memory or solid state drive (SSD) 218,which provides nonvolatile storage of computer-readable instructions,data structures, program modules and other data for the processor-baseddevice 204. Although not depicted, the processor-based device 204 canemploy other non-transitory computer- or processor-readable media, forexample a hard disk drive, an optical disk drive, or memory card mediadrive.

Program modules can be stored in the system memory 208, such as anoperating system 230, one or more application programs 232, otherprograms or modules 234, drivers 236 and program data 238.

The application programs 232 may, for example, include panning/scrolling232 a. Such panning/scrolling logic may include, but is not limited tologic that determines when and/or where a pointer (e.g., finger, stylus,cursor) enters a user interface element that includes a region having acentral portion and at least one margin. Such panning/scrolling logicmay include, but is not limited to logic that determines a direction anda rate at which at least one element of the user interface elementshould appear to move, and causes updating of a display to cause the atleast one element to appear to move in the determined direction at thedetermined rate. The panning/scrolling logic 232 a may, for example, bestored as one or more executable instructions. The panning/scrollinglogic 232 a may include processor and/or machine executable logic orinstructions to generate user interface objects using data thatcharacterizes movement of a pointer, for example data from atouch-sensitive display or from a computer mouse or trackball, or otheruser interface device.

The system memory 208 may also include communications programs 240, forexample a server and/or a Web client or browser for permitting theprocessor-based device 204 to access and exchange data with othersystems such as user computing systems, Web sites on the Internet,corporate intranets, or other networks as described below. Thecommunications program 240 in the depicted embodiment is markup languagebased, such as Hypertext Markup Language (HTML), Extensible MarkupLanguage (XML) or Wireless Markup Language (WML), and operates withmarkup languages that use syntactically delimited characters added tothe data of a document to represent the structure of the document. Anumber of servers and/or Web clients or browsers are commerciallyavailable such as those from Mozilla Corporation of California andMicrosoft of Washington.

While shown in FIG. 2 as being stored in the system memory 208, theoperating system 230, application programs 232, other programs/modules234, drivers 236, program data 238 and server and/or browser 240 can bestored on any other of a large variety of non-transitoryprocessor-readable media (e.g., hard disk drive, optical disk drive, SSDand/or flash memory).

A user can enter commands and information via a pointer, for examplethrough input devices such as a touch screen 248 via a finger 244 a,stylus 244 b, or via a computer mouse or trackball 244 c which controlsa cursor. Other input devices can include a microphone, joystick, gamepad, tablet, scanner, biometric scanning device, etc. These and otherinput devices (i.e., “I/O devices”) are connected to the processor(s)206 through an interface 246 such as a touch-screen controller and/or auniversal serial bus (“USB”) interface that couples user input to thesystem bus 210, although other interfaces such as a parallel port, agame port or a wireless interface or a serial port may be used. Thetouch screen 248 can be coupled to the system bus 210 via a videointerface 250, such as a video adapter to receive image data or imageinformation for display via the touch screen 248. Although not shown,the processor-based device 204 can include other output devices, such asspeakers, vibrator, haptic actuator or haptic engine, etc.

The processor-based device 204 operates in a networked environment usingone or more of the logical connections to communicate with one or moreremote computers, servers and/or devices via one or more communicationschannels, for example, one or more networks 214 a, 214 b. These logicalconnections may facilitate any known method of permitting computers tocommunicate, such as through one or more LANs and/or WANs, such as theInternet, and/or cellular communications networks. Such networkingenvironments are well known in wired and wireless enterprise-widecomputer networks, intranets, extranets, the Internet, and other typesof communication networks including telecommunications networks,cellular networks, paging networks, and other mobile networks.

When used in a networking environment, the processor-based device 204may include one or more network, wired or wireless communicationsinterfaces 252 a, 256 (e.g., network interface controllers, cellularradios, WI-FI radios, Bluetooth radios) for establishing communicationsover the network, for instance the Internet 214 a or cellular network.

In a networked environment, program modules, application programs, ordata, or portions thereof, can be stored in a server computing system(not shown). Those skilled in the relevant art will recognize that thenetwork connections shown in FIG. 2 are only some examples of ways ofestablishing communications between computers, and other connections maybe used, including wirelessly.

For convenience, the processor(s) 206, system memory 208, and networkand communications interfaces 252 a, 256 are illustrated as communicablycoupled to each other via the system bus 210, thereby providingconnectivity between the above-described components. In alternativeembodiments of the processor-based device 204, the above-describedcomponents may be communicably coupled in a different manner thanillustrated in FIG. 2 . For example, one or more of the above-describedcomponents may be directly coupled to other components, or may becoupled to each other, via intermediary components (not shown). In someembodiments, system bus 210 is omitted and the components are coupleddirectly to each other using suitable connections.

FIG. 3 is a high level flow diagram of a method 300 of operating a TDCsystem in accordance with the techniques of the present disclosure. Themethod 300 may, for example, be performed by TDC system 100 of FIG. 1 .

The method 300 begins at 302, wherein at least one processor of the TDCsystem receives labeled training data. As discussed above, the labeledtraining data may include data for a plurality of individuals thatindicates whether each of the individuals has one or more of a pluralityof mental health disorders, such as, but not limited to, schizophrenia,bipolar disorder, or attention deficit and hyperactivity disorder(ADHD). For each of the individuals, the labeled training data alsoincludes item-level responses to multiple self-administered mentalhealth questionnaires (e.g., five questionnaires, 10 questionnaires, 20questionnaires). In some examples, the training data includes video dataand audio data recorded while each of the plurality of individuals readtext from a digital display. In some examples, the video data isprocessed to identify portions of the video data comprising the face ofthe individual, and the audio data is processed to identify soundsrepresenting the voice of the individual

In at least some implementations, the labeled training data comprises adataset that is publicly-available from the UCLA Consortium forNeuropsychiatric Phenomics, which focused on the understanding of thedimensional structure of memory and cognitive control functions in bothhealthy individuals (130 subjects) and individuals diagnosed as havingneuropsychiatric disorders including schizophrenia (50 subjects),bipolar disorder (49 subjects), and ADHD (43 subjects) afteradministration of the Structured Clinical Interview for DSM Disorders bymental health professionals.

All participants provided item-level responses to multipleself-administered mental health questionnaires. In the exampleembodiment discussed herein, individuals' responses to a total of 578individual questions were used. The questions were obtained from thefollowing scales/questionnaires: Hopkins Symptom Checklist (HSCL); AdultSelf-Report Scale v1.1 Screener (ASRS); Barratt Impulsiveness Scale(BIS-11); Dickman Functional and Dysfunctional Impulsivity Scale;Multidimensional Personality Questionnaire (MPQ)—Control sub scale;Impulsiveness, Venturesomeness and Empathy Scale (WE); Scale for Traitsthat Increase Risk for Bipolar II Disorder; Golden & Meehl's Seven MMPIItems Selected by Taxonomic Method; Hypomanic Personality Scale (HPS);Chapman Scales (Perceptual Aberrations, Social Anhedonia, PhysicalAnhedonia); and Temperament and Character Inventory (TCI). It should beappreciated that in other implementations, one or more differentscales/questionnaires may be used, or various combinations of one ormore scales/questionnaires.

At 304, at least one processor of the TDC system trains an initialmachine learning classifier or model using the labeled training data.For example, in at least some implementations, k-fold cross-validation(e.g., 10-fold cross-validation) with logistic regression is used toclassify healthy control (HC) individuals from patients diagnosed withone or more mental health disorders base on the scores of the 578individual questions. Using each of the individual questions, the systemwas operative to classify subjects as either “HC” or “Patient” with amean accuracy of 79%. FIG. 4 is a graph 400 that shows the receiveroperating characteristics (ROC) curve, another evaluation metric, whichhad a mean area under the curve (AUC) of 0.88 (max 1).

At 306, the at least one processor of the TDC system extracts or obtainsa measure of feature importance for the 578 questions or features fromthe regression coefficients.

At 308, in order to examine if shortening the list of questions couldprovide comparable classification ability, the TDC system generates aseries of models, also referred to herein as subset ML models,sequentially adding in features in order of importance, starting withthe most important feature. For example, a first subset ML model mayinclude only the most important feature, a second subset ML model mayinclude the two most important features, a third subset ML model mayinclude the top three most important features, etc.

At 310, at least one processor of the TDC system may evaluate theperformance of at least some of the generated subset ML models. FIG. 5is a graph 500 that shows the area under the ROC curve for the subset MLmodels that include from one feature (i.e., the most important feature)up to each of the features. FIG. 6 is a graph 600 that shows an accuracyscore for each of the subset ML models.

It was found that classifier performance across different subsets ofquestions (i.e., questions 1 through 578) varied on AUC from 0.8 to0.97. Further, it was found that only 17 features are needed for anaccuracy of 91% and 0.95 AUC. This indicates that more features are notnecessarily better in a classifier-based screen. Notably, the top 17features included a disproportionate number of questions regardingpersonality and temperament with additional questions on impulsivity,mood, and mania. In an example embodiment, the 17 top features orquestions include the following questions, in order starting with themost important feature:

-   -   (1) “I like to please other people as much as I can” (tci28t);    -   (2) “There are often times when I am so restless that it is        impossible for me to sit still” (chaphypo8);    -   (3) “My mood often changes, from happiness to sadness, without        my knowing why” (bipolarii1);    -   (4) “Although there are things that I enjoy doing by myself, I        usually seem to have more fun when I do things with other        people” (chapsoc8);    -   (5) “I am more sentimental than most people” (tci55t);    -   (6) “I love to excel at everything I do” (tci72p);    -   (7) “People consider me a rather freewheeling and spontaneous        person” (mpq243);    -   (8) “I feel that I never really get all that I need from people”        (bipolarii26);    -   (9) “In unfamiliar surroundings, I am often so assertive and        sociable that I surprise myself” (chaphypo7);    -   (10) “I like to think about things for a long time before I make        a decision” (tci61t);    -   (11) “Sometimes ideas and insights come to me so fast that I        cannot express them all” (chaphypo5);    -   (12) “I have many hobbies” (dicks);    -   (13) “I like to keep my problems to myself” (tci68t);    -   (14) “It is difficult for me to keep the same interests for a        long time because my attention often shifts to something else”        (tci35t);    -   (15) “How often do you have trouble wrapping up the final        details of a project, once the challenging parts have been done”        (finaldetail);    -   (16) “I like to go slow in starting work, even if it is easy to        do” (tci189p); and    -   (17) “Usually I am more worried than most people that something        might go wrong in the future” (tci81t).        As would be readily understood by one skilled in the art,        variations of these questions can be used in the disclosed        systems and methods as well. In various examples of the present        disclosure, some additional questions can be used,        replacement/alternate questions can be used, or some of the        questions can be omitted.

At 312, at least one processor of the TDC system selects one or more ofthe subset ML models as a trans-diagnostic classifier based on theperformance evaluation. For example, at least one processor of the TDCsystem may select the subset ML model that includes the above-listed top17 features as a trans-diagnostic classifier. The selectedtrans-diagnostic classifier may then be used to screen subjects acrossmultiple mental health disorders without physician involvement,optionally remotely and repeatedly, in a short period of time (e.g.,less than 5 minutes).

In at least some implementations, the above described techniques mayadditionally or alternatively be used to generate a specific screen forindividual diagnoses, for example, a screen that classifies Healthy vs.Schizophrenic, or a screen that classifies Healthy vs. Bipolar, etc.Similar to the trans-diagnostic screen discussed above, each of thesescreens may include an associated shortlist of questions determinedusing the techniques used herein that allow for quick-screening relativeto existing screening methods.

Exemplary Screening System

The present disclosure contemplates that a variety of systems can beused to perform various embodiments of the present disclosure. FIG. 7Apresents an exemplary system 700A, which can be configured to performvarious methods of the present disclosure, including methods 720, 730,740, 800, and 900 of FIGS. 7B, 7C, 7D, 8, and 9 , respectively. Inparticular, system 700A includes a display 702; a user 704; a camera706; a camera field of view 706 a; a user interface 708; a remotecomputing device 710; and a microphone 712.

The camera 706 captures visual data of an area in front of the camera(area 706 a) and transmits the visual data to the display 702 and theremote computing device 710. As shown in FIG. 7A, a user 704 sits in theview of the camera 706. In such an example, the camera 706 capturesfootage of the face of the user 704. In some examples, the camera 706can be configured to take live video footage, photographs, orimages/videos in non-visual wavelengths. In some examples, the camera706 is configured to start or stop recording based on instructions fromthe remote computing device 710 or a local processor or computingdevice. For instance, the application or program running the process maybe performed by a remote server, computing device, or a local processor.The camera 706 is communicatively coupled to the display 702 and theremote computing device 710 or a local computing device. In someexamples, a smartphone will perform each of these functions.

The user interface 708 is configured to receive input from a user 704.For example, the user interface 708 can be a keyboard, a touchscreen, amobile device, or any other device for receiving input, as known in theart. The user 704 enters data on the user interface 708 in response toprompts on the display 702. For example, the display 702 outputs aseries of mental health questions, and the user 704 inputs an answer toeach question on the user interface 708. The user interface 708 isconfigured to directly display the input on display 702 and isconfigured to relay the data to the remote computing device 710.

The microphone 712 is configured to receive auditory input, for example,from the user 704. The microphone is configured to start or stoprecording based on instructions from the remote computing device 710.The microphone is configured to transmit audio data to the remotecomputing device 710. In some examples, the microphone can be on auser's smart phone.

The display 702 is configured to receive data from the camera 706, theremote computing device 710, and the user interface 708. For example,the display 702 displays the visual data captured by the camera 706. Inanother example, the display 702 displays input received from the userinterface. The display 702 is directly coupled to the camera 706 and themicrophone 712 in some examples; in other examples, the camera 706 andthe microphone 712 send their data to the remote computing device 710,which then processes the data and instructs the display 702 according tothe processed data. In other examples, the display 702 displays datareceived from the remote computing device 710. Exemplary data from theremote computing device 710 includes questions from a mental healthquestionnaire, answer boxes, answer options, answer data, a mentalhealth indicator, or any other information. In some examples, thedisplay 702 is on a smart phone.

The present disclosure also contemplates that more than one display 702can be used in system 702, as would be readily contemplated by a personskilled in the art. For example, one display can be viewable by the user704, while additional displays are visible to researchers and not to theuser 704. The multiple displays can output identical or differentinformation, according to instructions by the remote computing device710.

A remote computing device 710 can be communicatively coupled to adisplay 702, a camera 706, a user interface 708, and a microphone 712.For example, the communication can be wired or wireless. The remotecomputing device 710 is configured to perform any methods ascontemplated according to FIGS. 7B-9 (discussed further below). Theremote computing device 710 can process and/or store input from thedisplay 702, the camera 706, the user interface 708, and the microphone712.

In some examples, system 700 can be a user 704 with a unitary device,for example, a smart phone. The smart phone can have a display 702, acamera 706, a user interface 708, a computing device 710, and amicrophone 710. For example, the user 704 can hold the smart phone infront of his or her face while reading text on the display 702 andresponding to the mental health questionnaires. Referring briefly toFIGS. 10A-10B, an exemplary interface design is shown. Similar labelsare used for corresponding elements to FIG. 7A. FIG. 10A shows a screen1000A displaying text for a user to read, and FIG. 10B shows a screen1000B displaying a user's face as video data is being recorded. FIGS.10A-10B demonstrate how the disclosed system and methods can beperformed on a local device, with ease of access for the user.

Test Application for Voice/Facial Recognition During Screening

FIG. 7B shows an exemplary methodology 700B, according to an exemplaryimplementation of the present disclosure. Methodology 700B provides atest for a user and can be provided for by the system 700A, as discussedabove with respect to FIG. 7A.

Methodology 700B provides for, at step 720, controlling execution andtermination of a test application. The test application can be asoftware application stored on a computing device (e.g., the remotecomputing device 710 of FIG. 7A). Step 720 provides for executing thetest application upon receiving and indication to initiate a test. Insome examples, the indication comes from a user interface (e.g., theuser interface 708 of FIG. 7 a ) communicatively coupled to thecomputing device.

Step 720 provides for executing the test application until the computingdevice receives an indication to stop the test. In some examples, thisindication comes from the user interface. In some examples, theindication to stop the test includes determining, by the computingdevice, that the user's face is not within an image captured by acamera.

While the test is being executed according to step 720, methodology 700Bproceeds to step 721. Step 721 provides for displaying a series ofquestions. An exemplary series of questions includes questions frommental health questionnaires, and includes both text and answers foreach question.

While the test is being executed according to step 720, methodology 700Bcan provide for step 722. Step 722 provides for displaying live videodata. In some examples, live video data is collected from a camerapositioned to capture an image in front of a display (e.g., camera 706capturing visual data of user 704 positioned in front of the display702, as shown in FIG. 7A). In some examples, live video data is recordedand then displayed at a display; in other examples, live video data issimultaneously recorded and displayed. The display can be facing theuser.

While the test is being executed according to step 720, methodology 700Bcan provide for step 723. Step 723 provides for recording test videodata and test audio data (e.g., from camera 706 and microphone 712 ofFIG. 7A). In some examples, the audio data and the video data arerecorded in segments corresponding to the display of questions at step722; in others examples, the data is collected in an un-interruptedstream while the questions are presented at step 722. In some examplesof step 723, the video and audio data is pre-processed according tomethodology 730 of FIG. 7C.

In some examples, a microphone (e.g., microphone 712 of FIG. 7A) recordsaudio data upon determining, by the computing device, that the user isspeaking. In some examples, the microphone stops recording audio datawhen the computing device determines that the user is not speaking.

While the test is being executed according to step 720, methodology 700Bcan provide for step 724. Step 724 provides for receiving answers foreach of the series of questions (the questions provided for in step721). The answers are received at a user interface. In some examples,the answers include selection of a multiple choice question, a textualresponse, or any other user input as contemplated by one skilled in theart.

While the test is being executed according to step 720, methodology 700Bcan provide for step 725. Step 725 provides for processing the answersreceived at step 724 and the test video data and the test audio datarecorded at step 723. In some examples, the processing is performed at acomputing device using a machine learning model and outputs a mentalhealth indication of the user. In some examples of the presentdisclosure, step 725 performs processing of the answers, the test videodata, and the test audio data as discussed further below with respect tomethod 740 of FIG. 7D.

In some examples, the output mental health indication identifies alikelihood of the user having any one of several mental healthdisorders. The mental health disorders include a neuropsychiatricdisorder, schizophrenia, and a bipolar disorder. In some examples, themental health indication identifies whether the user is a patient or ahealthy control.

Steps 721, 722, 723, 724, and 725 of FIG. 7B can occur sequentiallyafter the test application is initiated in step 720. In some examples ofmethodology 700B, steps 721, 722, 723, 724, and 725 occur simultaneouslyand/or in any combination. In some examples of methodology 700B,portions of steps 721, 722, 723, 724, and 725 or any subsets of steps721, 722, 723, 724, and 725 are repeated or omitted according toinstructions from a remote computing device. Therefore, the presentdisclosure contemplates that any combination of the above description ofsteps 720, 721, 722, 723, 724, and 725 can be used in an embodiment ofthe present disclosure, as readily contemplated by one skilled in theart.

Referring now to methodology 730 of FIG. 7C, an exemplary methodology isshown for preprocessing audio and visual data, according to variousembodiments of the present disclosure. In step 731, methodology 730provides for receiving test video data and test audio data. In someexamples, the test video data and test audio data are recorded accordingto methodology 700B of FIG. 7B, or another embodiment of the presentdisclosure.

Step 732 provides for preprocessing the test video data to identifyvideo segments. Each video segment corresponds to one question in aseries of questions (e.g., questions from a test, as discussed withrespect to methodology 700B of FIG. 7B) and includes a time window; thetime window provides a duration of the video segment and a period oftime in the recorded data during which the video segment occurs. In someexamples, the time window includes any of: a start time, a stop time,and a duration length. In some examples, video segments are identifiedbased on instructions from a computing device according to whenquestions were displayed at a display.

Step 733 provides for preprocessing the test audio data to identifyaudio segments. Each audio segment corresponds to one question in theseries of questions and includes a time window; the time window is asprovided with respect to the time windows of step 732. In some examples,audio segments are identified based on instructions from a computingdevice according to when questions were displayed at a display. In someexamples, audio segments are identified based on a computing devicedetermining whether a user is speaking.

Step 734 provides for preprocessing the video segments of step 732 andthe audio segments of step 733 to identify overlapping time windows.Step 735 provides for outputting integrated audio and video segmentsbased on overlapping time windows. In some examples, the integratedaudio and video segments are stored on a remote computing device.

Referring now to methodology 740 of FIG. 7D, an exemplary methodology isshown for using a machine learning model to analyze input and output amental health indication, according to various embodiments of thepresent disclosure. In some examples, the machine learning model is anyof: a generalized linear model, a regression model, a logisticalregression model, and a supervised machine learning classificationmodel. In some examples, the machine learning model is any of the modelsand algorithms discussed further below.

In step 741, methodology 740 provides for receiving labeled trainingdata regarding mental health disorder status for a plurality ofindividuals. In some examples, the labeled training data identifieswhether each of the individuals has one or more mental health disorders.In some examples, the labeled training data includes audio and videodata recorded for each of the individuals (e.g., audio and video datarecording according to methodology 700B of FIG. 7B, or any otherembodiment of the present disclosure). The labeled training data canalso a selection of answers to mental health questionnaires. In someexamples, the labeled training data includes, for each individual, anindication of any of: whether the individual is healthy, whether theindividual has a general mental health issue, whether the individual hasone or more specific mental health disorders, whether the individual isat risk of developing a general mental health issue, or whether theindividual is at risk of developing one or more specific mental healthdisorders. In some examples, the labeled training data includesfunctional and/or physiological measurement data.

In step 742, methodology 740 provides for determining features from thelabeled training data of step 714. The features are determined accordingto any methods, as known in the art.

In step 743, methodology 740 provides for training an initial machinelearning model in a supervised manner, based on the features determinedin step 742. In some examples, training this initial machine learningmodel includes using k-fold cross-validation with logistic regression.

In step 744, methodology 740 provides for extracting importance measuresfor each of the features. These importance measures are selected basedon the trained initial machine learning model.

In step 745, methodology 740 provides for generating a plurality ofsubset machine learning models, based on the extracted importancemeasures of step 744. In step 746, methodology 740 provides forevaluating a classification performance of the generated subset machinelearning models from step 745. In some examples, each of the subsetmachine learning models includes a different selection of features. Insome examples, the subset machine learning models include only featureswith an importance measure above a threshold value.

In step 747, methodology 740 provides for selecting one of the subsetmachine learning models as a generalized linear learning model. Theselection is based on the classification performances as evaluated instep 746. The selected subset machine learning model includes a portionof the plurality of features determined from step 742. The portion offeatures is selected from features with an importance measure (asdetermined in step 744) above a threshold value. In some examples, morethan one subset machine learning model is selected.

In some examples of step 747, the threshold value is set so that atleast twenty features of the plurality of features determined in step742 have an importance measure above the threshold value. In someexamples, the threshold value is set to select a portion of between tenand twenty features.

In some examples of step 747, at least one of the subset machinelearning models is selected as a diagnostic classifier. The features ofthe diagnostic classifier are stored in a remote computing device forsubsequent use as a screening tool. In some examples, the diagnosticclassifier outputs a mental health indication. The mental healthindication can be any of: (1) identifying a user as healthy or as havinga general mental health issue, (2) identifying the user as healthy or ashaving a specific mental health issue, (3) identifying the user ashaving either a first specific mental health disorder or a secondspecific mental health disorder, and (4) identifying a risk ofdeveloping a mental health disorder for an individual.

The selected machine learning model can then be used to process any ofthe input data as provided for in the present disclosure. In someexamples, the features of the diagnostic classifier are used as ascreening tool to assess intermediate and/or end-point outcomes inclinical trial testing for treatment responses.

Overall, methods 720 of FIG. 7B and 730 of FIG. 7C provide algorithmswhich receive input in different modalities. Methodology 740 providesprocessing of the input from methods 720 and 730 to output an algorithmbased on features that have the highest predictive value (predictivevalue can be determined based on importance measures). For example,various embodiments of methods 720 and 730 receive mental healthquestionnaire data, voice data, and/or video data. Methodology 740receives all the input, determines features based on the input, anddetermines which of those features have the highest predictive value. Amachine learning model can be built which incorporates the features withthe highest predictive value.

Accordingly, the model, methodology, and model builder are especiallyvaluable and designed for efficiently combining features from multiplemodalities including various different scales instruments, video dataand audio data to build multi-modal models that can frequently be moreaccurate than single modality models. For instance, prior technologyrequired a new model to be built and trained for each new combination ofmodalities, which made it completely impractical to combine modalitiesefficiently, especially without introducing too much noise in theoutputs as prior technology cannot test features from various modalitiesto incorporate them into a single model. Particularly, in some examples,additional features may be less accurate, and therefore one key tocombining modalities is to incorporate the right features from eachmodality. The disclosed technology provides processes and models thatallow for their efficient testing and combination.

Furthermore, mental health screeners and models frequently benefit fromexamining more than one modality and may produce far superior accuracyin some examples. For instance, processing only answers from scalesbased questions may fail to capture the tone of voice and facialexpressions made while reading a statement—and other things like thespeed at which the statement is read. This features may be criticallyimportant to an assessment of the mental health of a patient, as forexample, a statement may be read in various tones of voice, or spokenwith the same tone of voice but with different facial expressions—allconveying different emotions and mental health status indicators.

This model can then be used as a diagnostic tool. For example,additional mental health questionnaire data, voice data, and/or videodata can be input into the model to determine a mental health indicationof a patient.

Therefore, the methods of the present disclosure provide machinelearning algorithms which can determine the features that are predictivefor various mental health disorders. For example, the machine learningmodel can determine a mental health indication related to a firstparticular mental health disorder relies on a first set of features;this first set of features can be from any input modality (e.g., adepression mental health indication can rely on tone of voice and facialexpression). The machine learning model can further determine that amental health indication related to a second particular mental healthdisorder relies on a second set of features; this second set of featurescan be from any input modality (e.g., an anxiety mental healthindication can rely on stuttering audio data or frequent self-referencesof the speaker). Accordingly, different features can be deterministicfor different mental health disorders; however, the same algorithm canbe used for different mental health disorders and for different inputdata. In some examples, the disclosed algorithm automatically adjustswhich mental health indications can be provided based on what input datais provided.

Application for Collecting Audio and Video Recording

FIG. 8 shows an exemplary methodology 800, according to an exemplaryimplementation of the present disclosure. Methodology 800 provides amethod of administering a test to a user and can be provided for by thesystem 700A, as discussed above with respect to FIG. 7A.

Methodology 800 provides for, at step 810, controlling execution andtermination of a test application. The test application can be asoftware application stored on a computing device (e.g., the remotecomputing device 710 of FIG. 7A). Step 810 provides for executing thetest application upon receiving and indication to initiate a test

While the test is being executed according to step 810, methodology 800can provide for step 820. Step 820 provides for recording test videodata and test audio data. The test video data can be captured by acamera (e.g., camera 706 of FIG. 7A) and the test audio data can becaptured by a microphone (e.g., microphone 712 of FIG. 7A).

While the test is being executed according to step 810, methodology 800can provide for step 830. Step 830 provides for displaying live videodata recorded by a camera and displaying text on the display for theuser to read. In some examples, the text includes a series of questionsor statements related to the user's mental health. The microphonecaptures audio data when the user reads the text aloud. In someexamples, the displayed text includes a series of questions from mentalhealth questionnaires, including question text and answer selections foreach question.

While the test is being executed according to step 810, methodology 800can provide for step 840. Step 840 provides for processing the set oftest video data recorded at step 820 to identify a face of the user anddetermine when the user's face is within the camera field of view. Insome examples, step 840 further comprises assigning a plurality ofpixels to the user's face and determining whether each of the pluralityof pixels of the user's face are within a frame captured by the camera.If the face is determined to be outside the frame captured by thecamera, step 840 provides for stopping the test. In other examples ofstep 840, a plurality of pixels are assigned to a boundary of the user'sfaced, and the step further provides for determining whether theboundary of the user's face is within a frame captured by the camera.The present disclosure further contemplates that any method can be usedto ensure that a user's face is within the camera's view, as known inthe art.

While the test is being executed according to step 810, methodology 800can provide for step 850. Step 850 provides for processing the testaudio data and the test video data to identify audio and video features.The audio and video features are stored in a memory of a computingdevice. In some examples of step 850, processing the test audio data andthe test video data is as provided for with respect to method 740 ofFIG. 7D above. In some examples, the audio and video features correspondto particular questions in the series of questions.

In some examples, before, during, or after step 850, methods 730 and 740of FIGS. 7C and 7D are applied to the test audio data and test videodata, as would be readily contemplated by one skilled in the art.

In some examples of methodology 800, steps 820, 830, 840, and 850 occursequentially after the test application is initiated in step 810. Insome examples, steps 820, 830, 840, and 850 occur simultaneously and/orin any combination. In some examples, portions of steps 820, 830, 840,and 850 or any subsets of steps 820, 830, 840, and 850 are repeated oromitted according to instructions from a remote computing device.Therefore, the present disclosure contemplates that any combination ofsteps 820, 830, 840, and 850 can be used in an embodiment of the presentdisclosure, as readily contemplated by one skilled in the art.

Interactive Test Application for Outputting a Screening Result

FIG. 9 shows an exemplary methodology 900, according to an exemplaryimplementation of the present disclosure. Methodology 900 provides amethod of administering a test to a user and can be provided for by thesystem 700A, as discussed above with respect to FIG. 7A.

Methodology 900 provides for, at step 910, receiving a set of answerdata. In some examples, the answer data includes answers from a user toa series of questions from mental health questionnaires.

Methodology 900 then provides for, at step 920, receiving a set of testvideo data and test audio data. In some examples, the test video dataand test audio data is recorded by a camera and a microphone (e.g.,camera 706 and microphone 712 of FIG. 7A). The test video data isrecorded during a test (e.g., the tests of methodologies 700B and 800 ofFIGS. 7B and 8 ) and includes the face of the user, while the user isreading text. For example, the text is displayed according tomethodology 800 of FIG. 8 . The set of test audio data is also recordedduring the test and represents the voice of the user, while the user isreading the text.

Step 930 of methodology 900 then provides for processing the set of testvideo data to output video features. Step 940 provides for processingthe set of test audio data to output audio features. In some examples,steps 930-940 are performed according to method 730 of FIG. 7C.

Methodology 900 further provides for, at step 950, processing the set ofanswer data, the set of video features, and the set of audio features tooutput a mental health indication. In some examples, step 950 isperformed as discussed above with respect to methodology 740 of FIG. 7D.

Machine Learning Implementation

Various aspects of the present disclosure can be performed by amachine-learning algorithm, as readily understood by a person skilled inthe art. In some examples, step 725 of FIG. 7B, methodology 740, step850 of FIG. 8 and step 950 of FIG. 9 can be performed by a supervised orunsupervised algorithm. For instance, the system may utilize more basicmachine learning tools including 1) decision trees (“DT”), (2) Bayesiannetworks (“BN”), (3) artificial neural network (“ANN”), or (4) supportvector machines (“SVM”). In other examples, deep learning algorithms orother more sophisticated machine learning algorithms, e.g.,convolutional neural networks (“CNN”), or capsule networks (“CapsNet”)may be used.

DT are classification graphs that match input data to questions asked ateach consecutive step in a decision tree. The DT program moves down the“branches” of the tree based on the answers to the questions (e.g.,First branch: Did the user pause before reading the question? yes or no.Branch two: Did the user stutter while reading the question? yes or no,etc.).

Bayesian networks (“BN”) are based on likelihood something is true basedon given independent variables and are modeled based on probabilisticrelationships. BN are based purely on probabilistic relationships thatdetermine the likelihood of one variable based on another or others. Forexample, BN can model the relationships between location data, timestamp data, previous alerts, and any other information as contemplatedby the present disclosure. Particularly, if a question type andparticular features of the user's auditory data are known, a BN can beused to compute the probability that a user has a particular mentalhealth disorder. Thus, using an efficient BN algorithm, an inference canbe made based on the input data.

Artificial neural networks (“ANN”) are computational models inspired byan animal's central nervous system. They map inputs to outputs through anetwork of nodes. However, unlike BN, in ANN the nodes do notnecessarily represent any actual variable. Accordingly, ANN may have ahidden layer of nodes that are not represented by a known variable to anobserver. ANNs are capable of pattern recognition. Their computingmethods make it easier to understand a complex and unclear process thatmight go on during predicting a mental health disorder based a varietyof input data.

Support vector machines (“SVM”) came about from a framework utilizing ofmachine learning statistics and vector spaces (linear algebra conceptthat signifies the number of dimensions in linear space) equipped withsome kind of limit-related structure. In some cases, they may determinea new coordinate system that easily separates inputs into twoclassifications. For example, a SVM could identify a line that separatestwo sets of points originating from different classifications of events.

Deep neural networks (DNN) have developed recently and are capable ofmodeling very complex relationships that have a lot of variation.Various architectures of DNN have been proposed to tackle the problemsassociated with algorithms such as ANN by many researchers during thelast few decades. These types of DNN are CNN (Convolutional NeuralNetwork), RBM (Restricted Boltzmann Machine), LSTM (Long Short TermMemory) etc. They are all based on the theory of ANN. They demonstrate abetter performance by overcoming the back-propagation error diminishingproblem associated with ANN.

Machine learning models require training data to identify the featuresof interest that they are designed to detect. For instance, variousmethods may be utilized to form the machine learning models, includingapplying randomly assigned initial weights for the network and applyinggradient descent using back propagation for deep learning algorithms. Inother examples, a neural network with one or two hidden layers can beused without training using this technique.

In some examples, the machine learning model can be trained usinglabeled data, or data that represents certain user input. In otherexamples, the data will only be labeled with the outcome and the variousrelevant data may be input to train the machine learning algorithm.

For instance, to determine whether particular mental health disorderfits the input data, various machine learning models may be utilizedthat input various data disclosed herein. In some examples, the inputdata will be labeled by having an expert in the field label the relevantregulations according to the particular situation. Accordingly, theinput to the machine learning algorithm for training data identifiesvarious data as from a healthy control or from a patient.

Computer & Hardware Implementation of Disclosure

It should initially be understood that the disclosure herein may beimplemented with any type of hardware and/or software, and may be apre-programmed general purpose computing device. For example, the systemmay be implemented using a server, a personal computer, a portablecomputer, a thin client, or any suitable device or devices. Thedisclosure and/or components thereof may be a single device at a singlelocation, or multiple devices at a single, or multiple, locations thatare connected together using any appropriate communication protocolsover any communication medium such as electric cable, fiber optic cable,or in a wireless manner.

It should also be noted that the disclosure is illustrated and discussedherein as having a plurality of modules which perform particularfunctions. It should be understood that these modules are merelyschematically illustrated based on their function for clarity purposesonly, and do not necessary represent specific hardware or software. Inthis regard, these modules may be hardware and/or software implementedto substantially perform the particular functions discussed. Moreover,the modules may be combined together within the disclosure, or dividedinto additional modules based on the particular function desired. Thus,the disclosure should not be construed to limit the present invention,but merely be understood to illustrate one example implementationthereof.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data (e.g., an HTML page) to aclient device (e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device). Datagenerated at the client device (e.g., a result of the user interaction)can be received from the client device at the server.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), an inter-network (e.g., theInternet), and peer-to-peer networks (e.g., ad hoc peer-to-peernetworks).

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations of the subjectmatter described in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a “data processing apparatus” on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

CONCLUSION

The various methods and techniques described above provide a number ofways to carry out the invention. Of course, it is to be understood thatnot necessarily all objectives or advantages described can be achievedin accordance with any particular embodiment described herein. Thus, forexample, those skilled in the art will recognize that the methods can beperformed in a manner that achieves or optimizes one advantage or groupof advantages as taught herein without necessarily achieving otherobjectives or advantages as taught or suggested herein. A variety ofalternatives are mentioned herein. It is to be understood that someembodiments specifically include one, another, or several features,while others specifically exclude one, another, or several features,while still others mitigate a particular feature by inclusion of one,another, or several advantageous features.

Furthermore, the skilled artisan will recognize the applicability ofvarious features from different embodiments. Similarly, the variouselements, features and steps discussed above, as well as other knownequivalents for each such element, feature or step, can be employed invarious combinations by one of ordinary skill in this art to performmethods in accordance with the principles described herein. Among thevarious elements, features, and steps some will be specifically includedand others specifically excluded in diverse embodiments.

Although the application has been disclosed in the context of certainembodiments and examples, it will be understood by those skilled in theart that the embodiments of the application extend beyond thespecifically disclosed embodiments to other alternative embodimentsand/or uses and modifications and equivalents thereof.

In some embodiments, the terms “a” and “an” and “the” and similarreferences used in the context of describing a particular embodiment ofthe application (especially in the context of certain of the followingclaims) can be construed to cover both the singular and the plural. Therecitation of ranges of values herein is merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (for example, “such as”) provided withrespect to certain embodiments herein is intended merely to betterilluminate the application and does not pose a limitation on the scopeof the application otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element essential tothe practice of the application.

Certain embodiments of this application are described herein. Variationson those embodiments will become apparent to those of ordinary skill inthe art upon reading the foregoing description. It is contemplated thatskilled artisans can employ such variations as appropriate, and theapplication can be practiced otherwise than specifically describedherein. Accordingly, many embodiments of this application include allmodifications and equivalents of the subject matter recited in theclaims appended hereto as permitted by applicable law. Moreover, anycombination of the above-described elements in all possible variationsthereof is encompassed by the application unless otherwise indicatedherein or otherwise clearly contradicted by context.

Particular implementations of the subject matter have been described.Other implementations are within the scope of the following claims. Insome cases, the actions recited in the claims can be performed in adifferent order and still achieve desirable results. In addition, theprocesses depicted in the accompanying figures do not necessarilyrequire the particular order shown, or sequential order, to achievedesirable results.

All patents, patent applications, publications of patent applications,and other material, such as articles, books, specifications,publications, documents, things, and/or the like, referenced herein arehereby incorporated herein by this reference in their entirety for allpurposes, excepting any prosecution file history associated with same,any of same that is inconsistent with or in conflict with the presentdocument, or any of same that may have a limiting affect as to thebroadest scope of the claims now or later associated with the presentdocument. By way of example, should there be any inconsistency orconflict between the description, definition, and/or the use of a termassociated with any of the incorporated material and that associatedwith the present document, the description, definition, and/or the useof the term in the present document shall prevail.

In closing, it is to be understood that the embodiments of theapplication disclosed herein are illustrative of the principles of theembodiments of the application. Other modifications that can be employedcan be within the scope of the application. Thus, by way of example, butnot of limitation, alternative configurations of the embodiments of theapplication can be utilized in accordance with the teachings herein.Accordingly, embodiments of the present application are not limited tothat precisely as shown and described.

REFERENCES

-   -   Kessler R C, et al. Short screening scales to monitor population        prevalences and trends in non-specific psychological distress.        Psychological Medicine 32:959-976 (2002)    -   Kessler R C, et al. Screening for serious mental illness in the        general population. Arch Gen Psychiatry 60:184-189 (2003)    -   Kessler R C, et al. The WHO World Mental Health (WMH) Surveys.        Psychiatrie (Stuttg) 6(1):5-9 (2009).    -   White P D, Rickards H, Zeman A Z J. Time to end the distinction        between mental and neurological illnesses. BMJ 344:e3454 (2012).

What is claimed is:
 1. A system for screening the mental health ofpatients, the system comprising: a display; a microphone; a camerapositioned to capture an image in front of the display; a memorycontaining machine readable medium comprising machine executable codehaving stored thereon instructions for performing a method of evaluatingthe mental health of a user; and a control system coupled to the memorycomprising one or more processors, the control system configured toexecute the machine executable code to cause the control system to:display, on the display, a set of text; record, by the camera, a set oftest video data which includes images of the user while reading the setof text aloud; record, by the microphone, a set of test audio data whichincludes sounds of the user while reading the set of text aloud; processthe video data to output a set of video features comprising facialexpressions of the user while reading the set of text aloud; and processthe audio data to output a set of audio features comprising tone ofvoice of the use while reading the set of text aloud; output a mentalhealth indication of the user by processing, using a machine learningmodel, the set of video features and the set of audio features, whereinthe machine learning model was generated by: receiving labeled trainingdata for a plurality of individuals, the labeled training datacomprising audio and video data recorded for each of the plurality ofindividuals; determining a plurality of features from the labeledtraining data; training an initial machine learning model using acombination of the audio data and the video data of the labeled trainingdata; using the trained initial machine learning model to generate aplurality of subset machine learning models based on the plurality offeatures; and selecting at least one of the subset machine learningmodels as the machine learning model.
 2. The system of claim 1, whereinthe control system is further configured to execute the machineexecutable code to cause the control system to: execute a testapplication, by the control system, upon receiving, from the userinterface, an indication to initiate a test; and terminate the testapplication upon receiving, by the control system, an indication to stopthe test, wherein the indication to stop the test application comprisesa determination, by the control system, that a user face is not withinan image captured by the camera.
 3. The system of claim 1, whereinrecording, by the microphone, further comprises: initiating therecording upon determining, by the control system, that the user isspeaking.
 4. The system of claim 1, wherein the control system isfurther configured to execute the machine executable code to cause thecontrol system to: receive the set of test video data and the set oftest audio data; preprocess the received set of test video data toidentify a plurality of video segments, each video segment correspondingto one phrase in the set of text and comprising a time window; andpreprocess the received set of test audio data to identify a pluralityof audio segments, each audio segment corresponding to one question in aseries of questions and comprising a time window.
 5. The system of claim4, wherein the control system is further configured to execute themachine executable code to cause the control system to: preprocess theplurality of audio segments and the plurality of video segments toidentify overlapping time windows; output a set of integrated audio andvideo segments based on the identified overlapping time windows.
 6. Thesystem of claim 1, wherein the machine learning model is selected fromthe group consisting of: a generalized linear model, a regression model,a logistical regression model, and a supervised machine learningclassification model.
 7. The system of claim 1, wherein the mentalhealth indication identifies a likelihood of the user having one of aplurality of mental health disorders, the plurality of mental healthdisorders comprising: a neuropsychiatric disorder, schizophrenia, and abipolar disorder.
 8. The system of claim 1, wherein the mental healthindication identifies whether the user is a patient or a healthycontrol.
 9. The system of claim 1, wherein selecting at least one of thesubset machine learning models as the machine learning model is based onclassification performance, wherein each of the subset machine learningmodels includes a subset of the plurality of features.
 10. The system ofclaim 9, wherein each of the subset machine learning models includesfeatures determined as having importance values above a predeterminedthreshold.
 11. The system of claim 9, wherein each of the subset machinelearning models includes a different combination of the plurality offeatures.
 12. The system of claim 1, wherein the set of text includes aseries of questions from mental health questionnaires, each questionincluding a prompt and an answer for the prompt.
 13. The system of claim12, wherein each of the plurality of features determined from thelabeled training data correspond to a question in the series ofquestions.
 14. The system of claim 13, wherein the labeled training dataincludes audio and video data recorded for each of the plurality ofindividuals while reading aloud the prompt and answer to the prompt foreach of the questions in the series of questions.
 15. A system forscreening the mental health of patients, the system comprising: adisplay; a microphone; a camera configured to capture an image in frontof the display; a memory containing machine readable medium comprisingmachine executable code having stored thereon instructions forperforming a method; and a control system coupled to the memorycomprising one or more processors, the control system configured toexecute the machine executable code to cause the control system toperform a test by: displaying a set of text on the display; recording,by the camera, a set of test video data which includes images of theuser while reading the set of text aloud; displaying, on the display,live video data recorded by the camera capturing the user; iterativelyprocessing the set of test video data during the test to: determinewhether each of a plurality of pixels of a face of the user are within aframe on the display; and stop the test in response to determining theface is outside the frame; processing the set of test video data toidentify video features; storing the video features in the memory,wherein the video features include facial expressions of the user whilereading the set of text aloud; and processing, using a machine learningmodel, the set of video features to output a mental health indication ofthe user, wherein the machine learning model was generated by: receivinglabeled training data for a plurality of individuals indicating whethereach of the plurality of individuals has one or more mental healthdisorders, the labeled training data comprising video data recorded foreach of the plurality of individuals while reading the set of textaloud; determining a plurality of features from the labeled trainingdata; training an initial machine learning model using a combination ofthe video data of the labeled training data; using the trained initialmachine learning model to generate a plurality of subset machinelearning models based on the plurality of features; and selecting atleast one of the subset machine learning models as the machine learningmodel.
 16. The system of claim 15, wherein iteratively processing theset of test video data further comprises: preprocessing a set of testaudio data with the test video data to identify overlapping timewindows; and outputting a set of integrated audio and video segmentsbased on the identified overlapping time windows.
 17. The system ofclaim 16, wherein the machine learning model is selected from the groupconsisting of: a generalized linear model, a regression model, alogistical regression model, and a supervised machine learningclassification model.
 18. A machine learning training system,comprising: at least one non-transitory processor-readable storagemedium that stores at least one of processor-executable instructions ordata; and at least one processor communicatively coupled to the at leastone non-transitory processor-readable storage medium, in operation, theat least one processor configured to: receive labeled training data fora plurality of individuals, the labeled training data furthercomprising: video data and audio data recorded while each of theplurality of individuals read predetermined text from a digital display,wherein the video data identifies portions of the video data comprisingthe face of the individual and the audio data identifies soundsrepresenting the voice of the individual while reading the text from thedigital display; process the audio data, and the video data to output aplurality of features; train an initial machine learning model using acombination of the audio data and the video data of the labeled trainingdata; use the trained initial machine learning model to generate aplurality of subset machine learning models based on the plurality offeatures; select at least one of the plurality of subset machinelearning models as a diagnostic classifier; and store ones of thefeatures corresponding to the diagnostic classifier in the at least onenon-transitory processor-readable storage medium.
 19. The machinelearning system of claim 18, wherein the diagnostic classifier isconfigured to output a mental health indication identifying anindividual as healthy or as having one of: (i) a general mental healthissue, and (ii) a specific mental health issue.
 20. The machine learningsystem of claim 18, wherein the diagnostic classifier is configured tooutput a mental health indication identifying an individual as havingeither a first specific mental health disorder or a second specificmental health disorder.
 21. The machine learning system of claim 18,wherein the diagnostic classifier is configured output a mental healthindication identifying a risk of developing a mental health disorder foran individual.
 22. The machine learning system of claim 18, wherein thelabeled training data further comprises: for each individual in theplurality of individuals, an indication of at least one of thefollowing: whether the individual is healthy, whether the individual hasa general mental health issue, whether the individual has one or morespecific mental health disorders, whether the individual is at risk ofdeveloping a general mental health issue, or whether the individual isat risk of developing one or more specific mental health disorders. 23.The machine learning system of claim 18, wherein training the initialmachine learning model further comprises using k-fold cross validationwith logistic regression.
 24. The machine learning system of claim 18,wherein the labeled training data further comprises at least one offunctional measurement data or physiological measurement data.
 25. Themachine learning system of claim 18, wherein the at least one processoris further configured to: use the features of the diagnostic classifieras a screening tool to assess at least one of intermediate or end-pointoutcomes in at least one clinical trial testing for treatment responses.